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ABSTRACT 



This report summarizes efforts by the state of New York to 



ensure the accuracy and integrity of data reported by school districts. The 
Office of the State Comptroller (OSC) has taken the lead in reviewing these 
data, focusing on the uses of attendance and enrollment data and making 
recommendations for improvement in state systems and local practices. Recent 
investigative reports described serious problems with educational data 
reported by the New York City school system, including incidents of fraud and 
abuse. These reports support findings in OSC audits and reinforce the need to 
improve procedures and internal controls for school-district data. School 
district data submissions provide the information used in the state's school 
district "report cards" and other statewide reports that evaluate school 
performance and provide accountability. The lack of adequate controls of 
these data is important as a matter of accountability, but also in regard to 
compulsory attendance laws. The report provides a summary of two audits and a 
followup review of district data, along with recommended changes in 
procedures. The recommendations encompass a broad range of data- integrity 
issues and provide a comprehensive framework for improvement. An appendix 
contains information on the impact of attendance and enrollment on state-aid 
payments. (RJM) 
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The accuracy and integrity of school district reported data, including test results, 
attendance, enrollment and many other items, is an important issue that the Office of the 
State Comptroller (OSC) has been working on for several years. This report summarizes the 
OSC efforts and audit recommendations for improvement in State systems and local - 
practices. It also examines the uses of attendance and enrollment data and provides a context 
within which to evaluate concerns being raised about school data. 

Overview of OSC Activities 

The Office of the State Comptroller is concerned with accountability and procedures 
statewide, and consequently has been pursuing school district data accuracy issues on a 
number of fronts. OSC audits have found problems and data inaccuracies in school districts 
visited around the State. By reviewing the practices and guidelines under which this data is 
collected, the OSC audits are able to recommend changes in local procedures and in SED’s 
policies which can improve the accuracy and integrity of the data. 

In addition to conducting a series of audits, OSC has been studying and revising its 
internal procedures for reviewing school aid vouchers that SED forwards for payment. OSC 
has moved from a transaction-based process to a risk-based audit approach, concentrating 
on high-risk and high-impact audit areas and using available electronic databases. The pre- 
and post-audit processes are also being more closely coordinated. In carrying out these 
responsibilities, OSC is working more actively with SED on their procedures. For example, 
OSC is working with SED staff on the triage process used to identify potential errors in 
school district data and OSC now participates in Regents Audit Committee meetings. 

Recent Investigative Reports 

In recent weeks, several highly publicized reports described serious problems with 
the educational data reported by the New York City schools system, including incidents of 
fraud and abuse. A report from the City’s Special Commissioner of Investigation (SCI) for 
the New York City School District described appalling incidents of misconduct by educators 
in which standardized test results were falsified or manipulated. Another report from the 
Governor’s Moreland Act Commission on New York City Schools found that attendance 
and enrollment figures had been erroneously or intentionally misreported. 

These investigative reports have findings similar to those in the OSC audits and 
reinforce the need to improve procedures and internal controls for school district data. 
Unfortunately, both of the recent reports examined only New York City schools and this 
may have led to a popular misconception that school district data accuracy is only an issue 
in New York City. This is not the case, as the OSC audits demonstrate. 
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Since the SCI is a City agency created explicitly to examine the New York City 
schools, naturally their reports cover only that system. However, in the case of the Moreland 
Act Commission, the Governor’s choice to limit the scope of the investigation entirely to 
New York City Schools was optional. A limited focus tends to distort the issue and does not 
allow for a comprehensive approach to correcting the problems. For example, the Moreland 
Act Commission, although it criticized the State Education Department for not having - 
sufficiently exercised its management and supervisory role over the attendance and 
enrollment data submitted, made no recommendations for changes in SED guidelines or 
practices. 

It has also been widely reported that the Moreland Act Commission found that 
overstated attendance data could be inflating State aid payments to New York City by 
hundreds of millions of dollars. Although the Moreland report discusses the potential impact 
of attendance on school aid, it may not be widely understood that the figures used are 
hypothetical or speculative. The Commission did not find the level of attendance 
misreporting that would drive the widely cited figures. 

Hypothetical calculations also ignore the reality that New York City’s school aid is 
annually decided in the State budget negotiations, and the formulas are then “worked 
backwards” to provide the agreed upon share. Thus — for New York City at least — it is 
very unlikely that changes in the input data would have any substantial real-world impact 
on aid. An appendix to this report describes the relationship between attendance and school 
aid, including the results of recent simulations prepared for OSC by SED and a discussion 
of the Moreland Act report’s school aid impact calculations. 

Uses of School District Data 

School district data submissions have a number of different uses. They provide the 
information included in school district “report cards” as well as other statewide reports used 
to evaluate school performance and provide accountability. The data also plays a role in 
local school district decisions, including funding allocations among school buildings (and 
among community school districts in New York City). Eventually, the data is also factored 
into the State school aid formula calculations, although there is generally a two- to three-year 
lag, and the many complexities in the school aid formulas tend to dampen the impact (see 
the appendix for a full discussion). 

Although the State aid impact of data inaccuracies may not be material or significant 
in most cases, it is nonetheless vitally important that State and local procedures ensure the 
integrity of data factoring into aid claims and performance reporting. In New York City and 
other districts, this data is also used to drive local budgetary allocations which are 
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immediate, direct and more likely to provide an incentive for misreporting than any potential 
State school aid impact. 

The lack of adequate controls and potential unreliability of school attendance data is 
important generally as a matter of accountability, but even more so when viewed in the 
context that New York State has compulsory attendance laws which local school districts - 
are charged with enforcing. The correlation between attendance rates and academic 
performance is very high, and schools cannot effectively improve attendance unless their 
reporting systems are functioning well. Moreover, errors in attendance record-keeping, even 
when they may be immaterial in terms of school aid payments, are none the less vitally 
important from the standpoint of a parent concerned that their school know the whereabouts 
of their child during the day. 

It should also be pointed out that the complexity of the current school finance system 
has strained both State and local resources to the point where data errors routinely occur. 
This is not an excuse for errors, and especially any intentional misreporting. However, the 
complexity is an important contributing factor. If attendance were one of, say, five or ten or 
twenty-five items submitted by schools for their aid, arriving at very solid data would be 
much easier. However, attendance is now only one of nearly 300 data elements feeding into 
more than 20 major school aid formulas. Comptroller McCall has advocated simplification 
and reform of the school finance system for a number of reasons, most importantly to 
improve the overall equity and efficiency of the system, but better data quality control would 
be another benefit of reform. Under a simplified system SED would be able to provide better 
quality control and OSC’s review of SED vouchers would also be greatly simplified; 
resources would be freed up in both agencies, allowing for more effective reviews. 

OSC Audits 

Today with this report, two audits and a follow-up review on school district data are 
being released: 

• State Education Department — Accuracy of Selected School District Data (Report 
97-S-62): This audit addresses the accuracy of data submitted by six upstate school 
districts to the State Education Department (SED), as well as the relevant procedures, 
- internal controls and guidelines established by SED. It tested data and procedures 
related to on attendance and enrollments, student scores on standardized tests, 
participation in the federal free and reduced price lunch program, and school district 
finances. (Note: the audit recommendations went to SED in November 1998.) 
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• SED Accuracy of Selected School District Data Follow-Up Review (Report 99-F-48): 
A follow-up on the State Education Department’s record in implementing the audit 
recommendations made in the report listed above. 

• New York City Board of Education — Accuracy of Selected School District Data 
(Report 98-N-2): This audit corresponds to the audit covering six upstate school - 
districts, testing data and procedures in place within the New York City school 
system, which is also subject to SED reporting rules and guidelines. 

Although there is considerable overlap in topics and findings, the OSC audits are 
different in nature from the investigations carried out by the SCI and Moreland Act 
Commission. Investigations are generally expected to identify specific incidents of fraud or 
abuse. In contrast, audits are more often focused on evaluations of data and systems to test 
whether there are sufficient controls and procedures in place. Following is a summary of the 
major findings and recommendations of the OSC school data accuracy audits: 

• OSC examined attendance, enrollment, standardized test scores, participation in the 
federal free and reduced price lunch program, and financial statement data at six 
upstate school districts. Errors were found in all of these categories, with the 
exception of the financial data (which must be independently audited annually), and 
errors were found in most categories at each of the six upstate school districts 
visited. These errors were attributable to inadequate systems of internal control at 
both SED and the school districts. Similar errors were found in the schools visited in 
New York City. 

• To address these problems, OSC recommended that SED: 

— Convert to a risk-based approach to better identify districts and data elements 
that are more likely to contain inaccuracies; 

— Implement a system for periodic independent reviews of attendance data and 
reporting systems; 

— Provide guidance for school districts concerning procedures and necessary 
internal controls for reporting attendance; and, 

— Require that standardized tests are scored and reported to SED by someone 
independent of school districts, or at a minimum ensure that teachers do not 
score their own students’ tests. 

• SED has initiated actions to address these recommendations, although these actions 
generally constitute only partial implementation, at least to date. For example, 
regional workshops were held on attendance, but SED has not yet provided the 
additional guidance that OSC recommended be given to districts statewide. 
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However, SED has begun reviewing its attendance policies and guidelines through 
an internal task force and the Commissioner has announced a schedule under which 
new policies and guidelines will be considered and promulgated in the fall of 2000. 

• Regarding student test scores, SED officials indicated that they have begun 
implementing regional scoring and team scoring to minimize the danger of 
intentional mis-scoring. In January 1999, the Grades 4 and 8 English tests were 
regionally scored. Regents examinations as well as the other standardized tests will 
now be team scored. 

• For local school districts, OSC recommended that necessary training be provided to 
managers and staff to ensure they understand attendance and enrollment reporting 
systems and responsibilities; that procedures for administering the federal free and 
reduced price lunch program be reviewed to ensure applications were valid; and that 
duties for grading and reporting of test results be separated to the extent possible. 

• The New York City school system, by virtue of its size and organizational structure, 
was the topic of a distinct audit. Some of OSC’s findings specific to the City’s school 
system were that a duplicative and unofficial attendance system (the working class 
list, or WCL) should be eliminated to reduce confusion and poor control, and that test 
scoring and reporting procedures should be improved, including accounting for all 
test booklets and separating duties. The audit also noted that an erasure analysis, even 
with proper follow-up, does not address all risks. 

• The audits did not find fault with the “default to present” design feature in the 
attendance system used by New York City, as well as districts throughout the State. 
Many record-keeping systems (including those for attendance and employment) use 
an approach where absences are recorded as the exception and a student or employee 
is otherwise presumed present. Such a system design need not be considered flawed 
in itself if there are adequate reconciliation and verification procedures in place. 

These audit recommendations encompass a broad range of data integrity issues and 
provide a comprehensive framework for improvement. Together with the work the State 
Education Department has promised, the State should have a good basis for moving toward 
a system that ensures more reliable and accurate school district data. 
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Appendix: Impact of Attendance and Enrollment on State Aid Payments 

Attendance and enrollment are primary inputs for most of the State’s school aid 
formulas used to apportion aid. However, the formulas are very complex, include save- 
harmless provisions and caps on aid, and use pupil counts from a number of different years. 
Estimating the dollar impact of a given change in pupil attendance data is therefore a very - 
complex operation in which different assumptions may yield very different results. 

Most of the “pupil counts” used in aid formulas are attendance-based, although 
enrollment is also used, and generally these counts are used on a lagged basis, and/or consist 
of averages of several years’ data. A single year’s school aid calculation therefore depends 
on both attendance and enrollment counts covering at least three school years. Myriad other 
factors in addition to attendance are involved in the calculations, including school district 
wealth, expenditures, and placement of special needs children. There are about 20 major aid 
formulas, many of which are altered annually. Several of the largest formulas are governed 
by a “transition adjustment,” which provides save-harmless protection and limits aid 
increases. All of these factors tend to dampen the impact of changes in attendance or 
enrollment on aid. Moreover, if aid is capped or a school district is on save-harmless, a 
change in pupil counts may have no effect, at least on the formulas so governed. 

The impact of any misreported pupil counts on school aid payments would vary 
substantially among school districts and even for a single school district, depending upon 
the year it occurs. This happens because the relative impact depends on many factors in each 
school district’s aid calculations and the other input data changes every year as do the aid 
formulas themselves — meaning there is no standard relationship between pupils and aid. 

Because of these methodological complexities, there is no simple answer to a 
seemingly simple question such as: what impact would a 2 percent overstatement of 
attendance have on aid? The answer would depend on the school district, the year, and 
whether one assumes that previous years’ pupil counts and aid calculations are also affected. 
Depending on the assumptions used, the answer could vary tremendously. 

SED Simulations of the Impact of Attendance/Enrollment Changes 

In response to the findings in OSC Audit Report 97-S-62, SED in November 1998 
estimated the dollar impact of the attendance errors and discrepancies found in that report. 
The audit had found attendance to be overstated by between 0.58% and 1 .5% at three upstate 
school districts, and to be understated by 12.2% at another district. This simulation, 
however, was performed only for the single largest aid category, “operating aid,” and was 
based on changes in only one year’s attendance data (the audit covered a single year’s data). 
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SED’s analysis concluded that only a minimal impact in aid would occur because, although 
attendance data directly affects the calculation of operating aid, the aid received is impacted 
by the “transition adjustment” and most districts do not receive the amount initially 
calculated in the formula. 

The SED simulation showed an almost negligible impact — the percentage decrease - 
in aid for districts with overstated attendance was between 0.01% and 0.03% (hundredths 
of percent) and the district with understated enrollment would not, according to this 
simulation, receive additional funds because its aid was capped. This simulation, however, 
was based only on operating aid, and thus did not reflect any impact in other aid formulas 
which are not governed by the cap in aid imposed by the transition adjustment. 

In response to a more general request from OSC in February 1999, SED provided a 
very different estimate of the impact on school aid for districts across the State if their 
attendance and enrollment were overstated. Unlike the previous analysis, this simulation 
covered all of the major computerized aid formulas. It was, however, entirely hypothetical 
and was intended to assist OSC in evaluating the potential impact of attendance 
inaccuracies. For purposes of the exercise, SED assumed that all of the pupil counts 
factoring into aid for the 1998-99 school year (17 attendance-based and enrollment counts) 
were overstated by 5 percent. In providing the simulation, SED commented that such an 
assumption was based on a highly unlikely if not impossible scenario (all types of attendance 
and enrollment overstated by an identical proportion in data covering three years). Overall, 
the SED simulation showed a generally dampened effect — statewide, 5 percent less pupils 
would mean 2.3 percent less aid. 



Impact of an assumed 5% reduction in Attendance and Enrollment 


Reduction in Aid: 


# of School Districts 


Percent of Districts 


greater than 5% 


12 


2% 


3 - 5% 


74 


11% 


2 - 3% 


125 


18% 


1 - 2% 


285 


42% 


less than 1% 


186 


27% 



This result varied substantially among school districts, however, as shown above. A 
majority of districts, according to the simulation, would experience an aid impact of less 
than 2% in response to the hypothetical 5% reduction in attendance and enrollment. Still, 
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there were many significant simulated impacts, with 31% of districts showing an impact 
exceeding 2%, and for 2% of the districts the impact even exceeded the percentage reduction 
of attendance assumed (which is not unexpected since the pupil counts may affect some 
formulas in a multiplicative manner). 

The significant, although dampened impact of attendance and enrollment data on - 
State school aid shown in this simulation is helpful to understanding the relationship. The 
complexity of the current school aid calculations, and particularly the caps in aid, tend to 
dampen the relationship, but it would be a mistake to conclude that there is none, or that it 
will always be negligible. 

It is hard to say what the “right” approach is to provide an appropriate description of 
the impact attendance has on State school aid. Actual audit corrections to school aid data 
will usually be limited to a particular year’s data, and given the current complex operation 
of the formulas, it is not likely that material aid impacts will occur. On the other hand, if the 
current system of attendance reporting were resulting in continuing and substantial 
misreporting, the impact might be greater. 

Legislative Alteration of the Aid Formulas 

It should be understood, however, that the major problem with hypothetical 
simulations such as those described above, is that the formulas are changed each year by the 
State Legislature. These annual modifications to the formulas are designed to produce a 
specific increase in aid with a particular distribution among school districts. Thus, if the data 
had been different in any given year, different aid formulas would have been enacted. This 
consideration has particular importance for making statements about what impact attendance 
inaccuracies would have on New York City, because the aid provided to the City’s schools 
is a focus of each year’s State budget negotiations. It is well known that the formulas are 
annually “worked backwards” until the politically negotiated “share” for the City schools 
is hit in the calculations. In this context, the data feeding into the school aid formulas for 
New York City is really of no practical consequence whatsoever — the City will get the 
negotiated share of aid regardless of what data they report. 

The Moreland Act Commission Estimate 

The Moreland Act Commission report has been widely publicized, as has an estimate 
it contains, to the effect that overstatement of attendance for New York City may have an 
impact of $150 million to $300 million. However, the Moreland report’s estimate is based 
only on the hypothetical impact over a five-year period of a hypothetical overstatement of 
attendance of 2-3 percent. The specific incidents of incorrectly reported attendance cited in 
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the Moreland report do not add up to such an over-reporting, and the report neither estimates 
an overstatement of 2-3 percent, nor contains data supporting such an estimate. In fact, the 
largest example cited (the missing scan sheets) would indicate an attendance overstatement 
of approximately 0.07% (7 hundredths of a percent). 

The Moreland report’s discussion of the impact of attendance and enrollment on State 
aid is similar to that presented above. The problem is that the hypothetical discussion has 
been confused with the actual incidents of attendance misreporting described. Moreover, the 
report’s hypothetical example is constructed under the aggressive assumption that there is 
a continuing overstatement of attendance and enrollment for New York City over five years 
time. Without questioning the technical construction of the estimate, it is important to 
understand that the $150-300 million cited is the cumulative impact over five years, 
representing between 0.5% and 1% of the $23-30 billion in aid New York City would 
receive over such a period. 

On a practical basis, however, it is probably most important to understand that these 
hypothetical calculations are based on an assumed overstatement of attendance much larger 
than what has reportedly been found and do not take into account the legislative process 
which tailors the aid formulas to produce a specific aid result for New York City. 
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